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The hybrid brain-computer interface (BCI)'s multimodal technology enables precision 
brain-signal classification that can be used in the formulation of control commands. In the 
present study, an experimental hybrid near-infrared spectroscopy-electroencephalography 
(NIRS-EEG) technique was used to extract and decode four different types of brain signals. 
The NIRS setup was positioned over the prefrontal brain region, and the EEG over the 
left and right motor cortex regions. Twelve subjects participating in the experiment were 
shown four direction symbols, namely, "forward," "backward," "left," and "right." The 
control commands for forward and backward movement were estimated by performing 
arithmetic mental tasks related to oxy-hemoglobin (HbO) changes. The left and right 
directions commands were associated with right and left hand tapping, respectively. The 
high classification accuracies achieved showed that the four different control signals can 
be accurately estimated using the hybrid NIRS-EEG technology. 
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INTRODUCTION 

Brain-computer interface (BCI) is a methodology that corre- 
lates brain activities with external devices. The recent research 
and trend have demonstrated the enormous potential of the 
BCI approach (Matthews et al., 2008; Nicolas-Alonso and 
Gomez-Gil, 2012; Ortiz-Rosario and Adeli, 2013). The material 
advances in the cutting-edge technology, moreover, has reduced 
the cost of BCI equipment. The BCI domain comprehends 
both invasive and non-invasive methods. Invasive methods such 
as electrical-corticography (ECoG), though showing promising 
signal-acquisition results, are not recommended, as they entail 
very significant risks. Non-invasive methods are much safer alter- 
natives in this regard (Min et al., 2010). 

The major non-invasive modalities include electroencephalog- 
raphy (EEG), magneto encephalography (MEG), functional mag- 
netic resonance imaging (fMRI), and functional near-infrared 
spectroscopy (fNIRS). Each has its own strengths and limitations; 
the selection of one over another for brain-imaging applications 
will rely on the cost of the equipment as well as the spatial and 
temporal resolution required for the given objective (Min et al., 
2010). 

EEG is a medical imaging technique that gauges brain activity 
by measuring, via metal electrodes positioned on the scalp, the 
voltage fluctuations on the scalp resulting from neurons' action 
potentials (Niedermeyer and Lopes da Silva, 1999; Rehan and 
Hong, 2012). The drawback of EEG is the poor spatial resolution 
that does not allow an accurate localization, that is, identification 
of the brain source signal (Ball et al., 2009). 

NIRS is another non-invasive brain-imaging technique that 
alternatively utilizes the near-infrared (NIR) spectrum of 
light (wavelength 600-1000 nm) to measure the hemodynamic 



response represented by oxygenated hemoglobin (HbO), deoxy- 
genated hemoglobin (HbR), cytochrome oxidase (CytOx) and 
water (H2O) concentration changes (Nagdyman et al., 2003; Irani 
et al., 2007; Bhutta et al., 2014). In most analyses, two hemo- 
dynamic variations due to brain activites are focused: increased 
oxygenation (resulting from the increased neural activity) and 
decreased deoxygenation (Matsuyama et al., 2009). Increased 
oxygen consumption in the course of performing increasingly dif- 
ficult mental tasks has been demonstrated (Verner et al., 2013). 
fNIRS has also shown the ability to detect the fast optical response 
(Hu et al., 2011), however the hemodynamic response is mostly 
used for analysis. 

EEG offers good temporal resolution (~0.05 s) but poor spa- 
tial resolution (~10mm), while fNIRS provides only moderate 
temporal resolution (~ls) and also moderately better spatial 
resolution (~5mm) (Nicolas-Alonso and Gomez-Gil, 2012). 
Another advantage of fNIRS to EEG is its robustness to noise 
(Waldert et al, 2012). The objective of a hybrid BCI (Pfurtscheller 
et al., 2010) is either to improve the classification accuracy or/ and 
to generate more control commands than the case of a single 
modality. The reason why Fazli et al. (2012) could improve the 
classification accuracy by using a hybrid EEG and fNIRS config- 
uration is that they used the union of two cases (i.e., detected by 
either EEG or fNIRS). This was possible because the window size 
for which the features are identified has been set to include both 
fNIRS and EEG data. Even for some cases that EEG could not 
detect, fNIRS could detect them. For motor execution, the aver- 
age classification accuracy by EEG alone was 90%, but EEG -|- 
HbR provided 93%, see Table 1 in Fazli et al. (2012). The previous 
studies on single modality have shown that the classification accu- 
racy for two commands using fNIRS was about 65% (Stangl et al.. 
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2013), and that for four commands using EEG (rotations and 
movements of the left/right wrists) was about 65% ( Vuckovic and 
Sepulveda, 2012). The objective in this paper, however, is to gen- 
erate more commands without losing the classification accuracy. 
In this paper, four commands will be generated: two from the 
prefrontal cortex and two from the motor cortex by configuring 
fNIRS and EEG in such a way that each control signal is generated 
from its associated brain region. In this way, a new command can 
be generated in every 0.6 s and the achieved classification accuracy 
was over 80%. 

The BCI techniques to control a wheel chair are diverse: eye 
movement and blinking based (Gneo et al., 2011; Lin and Yang, 
2012), emotions based (Fattouh et al., 2013), and event related 
and state control (Galan et al., 2008; Huang et al, 2012; Carlson 
and Millan, 2013). All these are reactive BCI, in which output 
from the brain is generated in reaction to an external stimulation. 
The novelty in this work is the proposition of a hybrid config- 
uration of EEG and fNIRS for active BCI, whose classification 
accuracy is over 80%. Using four brain tasks (left/right motor exe- 
cution, mental counting, and mental arithmetic), four commands 
were generated. In the proposed configuration, EEG electrodes 
are placed on the motor cortex region and NIRS optodes on 
the prefrontal cortex region. The left and right directions were 
decoded by tapping of the left or the right hand, and the mental 
arithmetic and the mental counting was used to decode back- 
ward and forward directions. The classification accuracies of the 
12 subjects justify that the proposed configuration is suiltable for 
BCI and direction decoding which can be used for the generation 
of control commands for movement executions for the patients 
suffering from lower-limb disorders. 
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FIGURE 1 I Experimental paradigm. One complete data set over the span 
of a minute, consisting of four rest periods, two task periods detected by 
NIRS from the prefrontal brain region and two motor execution periods 
detected from C3 and C4 regions of brain using EEG. 



METHODS 

PARTICIPANTS AND EXPERIMENTAL PARADIGM 

Twelve healthy volunteers (all male;10 right handed, two left 
handed; aged 24-34 years) participated in the experiment. The 
experiment was conducted under the declaration of Helsinki and 
consent was taken from the subjects priror to the start of exper- 
iment. The experiment was performed in a confined room to 
reduce disturbance from the environment. The subjects were sat 
in a comfortable chair with their arms on arm rests and instructed 
to relax. A screen nearly 70 cm away from the subjects was placed 
on which left, right, forward, and backward arrows were dis- 
played. On display of each arrow, a time marker starts at the 
bottom of screen indicationg the start and end of stimulus. For 
right and left directions, the subjects were asked to tap their asso- 
ciated hands 10 times during the time period shown on the screen 
for 10 s. For forward and backward directions, the subjects were 




FIGURE 2 I Optode location for EEG and NIRS. (A) 8 EEG electrodes 
placement over the Cz of the brain, (B) 12 channel locations on the 
prefrontal brain region using three sources and eight detectors. 
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asked to do mental counting for 10 s and arithmetic subtraction 
for 10 s. A training session was performed before the start to 
make the subjects familier with the paradigm. The total duration 
of each experiment for each subject was 5 min, divided into rest 
and activity periods. The time duration of each data sample was 
60 s. The initial 5 s of the experiment was the rest period, after 
which the subjects were shown left or right direction symbols 
and requested to physically tap their left or right hand accord- 
ingly, at a frequency of 1 Hz over a 10 s interval. The subjects 
were also instructed to increase the strength of tap for left hand 
for better discrimination of signals. The next 5 s was another rest 
period, subsequent to which the subjects were shown a forward 
or backward direction symbol for an interval of 10 s. The subjects 
were instructed to perform arithmetic subtraction, on a task sheet 
placed 25 cm away prior to the start of the experiment, upon the 
display of the backward direction symbol. For the mental arith- 
metic task, the subjects were asked to mentally perform a series 
of arithmetic calculations that were given on the sheet in a pseu- 
dorandom order. These calculations consisted of subtraction of a 
two-digit number (between 10 and 20) from a three-digit number 
throughout the task period with successive subtraction of a two- 
digit number from the result of the previous subtraction (e.g., 
695-19, 706-12, 894-15, etc). This mental activation period was 
followed by a 5-s rest period, after which the subjects were again 
shown the left or right direction symbol and instructed, once 
again, to tap their hand at a 1 Hz frequency over the time span 
of 10 s. If in a previous case the subjects were shown the left direc- 
tion symbol, the next time the indicated direction symbol was the 
opposite one. The directed hand activity was followed by 5 s rest, 
which was followed in turn by 10 s of mental activation. In this 
case as well, the next shown direction was reversed, according to 
which the subjects were asked to perform an arithmetic counting 



for indicated forward direction. For the mental counting task, the 
subjects were asked to mentally count down from number "99" 
backwards. For "Stop" the subjects were asked not to perform any 
activity. A check on EEG and NIRS data values was placed to dis- 
tinguish the activation and resting state by defining the baseline 
and activity. The movement command can only be produced if 
there is activity in any one of the four brain regions else the state 
should be termed as "Stop" state. Figure 1 shows one complete 
sample process; the second sample was taken immediately after 
the obtainment of the first. 

SENSOR CONFIGURATION 

Eight EEG electrodes were placed on the motor cortex region 
on the scalp and 12 channel NIRS was placed on the prefrontal 
brain region. The reason for using NIRS on the prefrontal cor- 
tex is because it can discriminate between two activities from the 
prefrontal region with high classification accuracies (Naito et al., 
2007; Power et al, 2010; Verner et al, 2013; Naseer et al., 2014) 
whereas the same cannot be done using EEG (Knyazev, 2013). 
Meanwhile NIRS signals are affected by dense hairs (Gervain 
et al., 2011) making EEG a better option for the detection of 
brain activities from the motor cortex region. Furthermore, if 
both modalities are positioned at the same brain location, they 
induce noise in each other thus reducing the strength of obtained 
signals for BCI (Safaie et al., 2013). Using the current setup, four 
signals were obtained thus enhancing the performance of NIRS 
by combining with EEG setup. 

DATA ACQUISITION 

The brain activities related to the mental tasks and motor exe- 
cutions were measured from the NIRS and EEG, respectively. 
Eight Ag/AgCl EEG electrodes were placed on C3, C4, T3, T4, 
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FIGURE 3 I System block diagram. The complete process from signal acquisition to control commands generation. 
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P3, P4, F3, and F4 locations according to the International 
10-20 system (Homan et al., 1987; Pivick et al, 1993; Jurcak 
et al., 2007), and the data were recorded by g-MOBIlab+ 
biosignal acquisition device (Christoph Guger, Austria) at a 
sampling rate of 256 Hz. The NIRS-System (DYNOT, NIRx 
Medical Technologies, USA) was used in this experiment with 
wave length of 760 and 830 nm, respectively. A total of three 
sources and eight detectors forming a combinational pair of 
12 channels were used in the experiment. This assembly was 
placed on Fpl and Fp2 regions of the brain and the optodes 
were placed in the way that they cover the whole prefrontal 
area in order to maximize the probability of locating the acti- 
vated region of brain. The sampling frequency used for the 
acquisition of NIRS signals was 1.81 Hz. Figure 2 shows the 
source-detector locations for the optode placement for EEG and 
NIRS. 

DATA ANALYSIS 

The fNIRS signals were obtained using the modified 
Beer-Lambert law (Coyle et al., 2007; Hu et al., 2010, 2012; 



Kamran and Hong, 2013; Naseer and Hong, 2013). 
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where A is the absorbance of light (optical density), /,„ is the inci- 
dent intensity of light, /out is the detected intensity of light, a is 
the specific extinction coefficient in (iM^'cm^', c is the absorber 
concentration in |xM, / is the distance between the source and 
detector in cm, d is the differential path-length factor, and r\ is the 
loss of light due to scattering. In order to remove noise from the 
hemodynamic response, different techniques are used (Santosa 
et al., 2013). In the present study, respiration- and pulse-related 




FIGURE 4 I Brain-signal acquisition: Each spike from the left or right motor cortex represents a hand tapping with the right or left hand, respectively. The 
concentration change in the prefrontal brain region reflects the change from rest to mental counting ("forward") and mental subtraction ("backward"). 
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noises were removed from the data using Gaussian low-pass fil- 
tering and wavelet transform (Ye et al, 2009; Hu et al., 2013). 
The P-band falling within the 12-30 Hz range was obtained by 
online band-pass filtering of the EEG signals (Zaepffel et al., 
2013; Kaiser et al., 2014). The epochs lasting 10 s were estimated 
by segmenting the recordings from -|- 1 to -|- 11 s relative to the 
onset of the tapping stimulus, thus yielding five epochs for each 
left and right hand activity (Delorme and Makeig, 2004; Subasi 
and Gursoy, 2010; Turnip et al, 2011; Turnip and Hong, 2012). 
Linear discriminant analysis was used as the classifier. The fea- 
tures in the case of EEG were the mean values of peak amplitudes 
of channels C3 and C4 whereas in the case of NIRS, the mean 
values of HbO and HbR were used (Lotte et al, 2007; Zhang 
et al., 2013; Naseer et al., 2014). An online analysis was performed 
on the data by downsampling EEG data to 1.82 Hz to synchro- 
nize with the NIRS sampling rate. For both modalities, one data 
sample was obtained approximately after 0.5 s, which was then 
processed at 250 Hz to obtain the control command. Figure 3 
shows the complete process for control command genereation 
for BCI. 



RESULTS 

The signals extracted from the left and right brain hemispheres 
(the C3 and C4 regions) and the frontal brain hemisphere (the 
Fpl and Fp2 regions) are shown in Figure 4. The signal from 
the motor cortex region reflects the action potential generated 
due to the firing of neurons when an motor execution task was 
performed. The signal from the prefrontal region is the hemo- 
dynamic change of HbO from the rest to mental execution 
period. 

The association between movement and rest state was devel- 
oped by taking the common rest state as "Stop" indication. This 
is also beneficial as using this methodology only one command 
can be generated at one time and thus reduces the chance of miss- 
classification. The movement command can only be produced if 
there is activity in any one of the four brain regions else the state 
should be termed as "Stop" state as shown in Figure 5 and Table 1 . 

The left- and right-signal classifications are shown in Figure 6. 
Similary forward and backward mental tasks are represented in 
Figure 7. Figures 5, 6 show changes of state of data with respect 
to time. The results indicate a significant difference between the 
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FIGURE 5 I Stop condition: Common rest state of the three signals is selected as stop command. The "Stop" command can only be generated if absolute 
rest is detected from three brain regions. 
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States. Table 2 lists the classification accuracies for all trials of the 
12 subjects. 

DISCUSSION 

The first objective of this study, which was achieved, was to detect 
four different classes of signal for generation of control com- 
mands for movement estimation suitable for BCI purposes. Two 
classes of signal were obtained by EEG using motor execution 
from the left and right motor cortex regions, and two classes are 

Table 1 | Brain activities of EEG and NIRS for different control 
commands. 



Brain activity 



EEG 

Left hand tapping 

Right hand tapping 

Rest 

Rest 

Rest 



NIRS 

Rest 
Rest 

Mental arithmetic 
Mental counting 
Rest 



Command 



Left 

Right 

Back 

Forward 

Stop 



obtained by NIRS from the prefrontal brain region using mental 
counting and arithmetic. These two tasks are simple tasks that 
are very easy to perform and can be detected from the prefrontal 
cortex. For the decoding application they serve their purpose 
well and have already been used in several BCI studies (Naito 
et al, 2007; Power et al, 2012; Naseer et al, 2014). Figure 4 
clearly shows that each tap recorded fi-om the left and right brain 
hemispheres due to the motor execution was recorded as a signal 
spike from the reference position. The time interval between each 
tap was maintained consistent at approximately 1 s. In order to 
differentiate between the left- and right-executed signals, each 
trial was separated by a time interval of 20 s. It was observed that 
the HbO concentration changes during the mental task began to 
increase 2 s after the subjects were prompted, and that the HbO 
level required almost 12 s to settle after the termination of that 
signal. Accordingly, the time gap between the two NIRS tasks was 
set at 20 s. 

The classification of left and right hand tapping is per- 
formed by increasing the magnitude of left hand tapping from 
the right hand tapping. It has been reported in jochumsen 
et al. (2013) and Robinson et al. (2013) that the neuronal 
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FIGURE 6 I EEG data classification for left and right control commands. (A) Shows the classification of trials for "Left" and "Stop" command signals for 
Subject 7 (B) Shows the classification of trials for "Right" and "Stop" command signals for Subject 6. 



Frontiers in Human Neuroscience 



www.frontiersin.org 



April 2014 1 Volume 8 j Article 244 | 6 



Khan et al. 



Hybrid NIRS-EEG brain-computer interface 



A X 10" 

20 



15 



Forward vs Stop 



10 



1 

'.> 5 
< 



-5 



1 1 1 1 1 
Trial separation 


1 1 1 


1 1 


X Stop 
V Forward 












▼ 


Forward 






Stop 




X V 

>^ X X 

1 1 1 1 1 


1 1 1 


X 

1 1 



2 0 40 60 80 1 00 120 140 

Time (seconds) 



160 



200 220 



0 04 



0 03 



o 

I 0.02 
■| 0 01 



Backward vs Stop 



1 1 

o 
o 


1 1 1 

cP 


Backvrard 


1 1 1 

oo Trial separation 
X Stop 
o Backward 

o 

o 

O- 






Stop 


X 


- xc o 




XX 




1 1 


1 1 1 


1 


1 1 1 



40 60 80 100 120 140 160 

Time f seconds'! 



180 2 00 220 240 



FIGURE 7 I IMIRS data classification for forward and backward control commands. (A) Shows the classification of trials for "Forward " and "Stop " 
command signals for Subject 3, (B) Shows the classification of trials for "Backward" and "Stop" command signals for Subject 7. 



response due to fast and slow movement is different and can 
be distinguished with high classification accuracy. Figures 6, 
7 show clearly that the four corresponding signals are sep- 
arable and, thus, utilizable for generation of BCI control 
commands. 

The observed classification accuracies for the 12 subjects sig- 
nificantly show that the proposed research is suitable for BCI 
purpose. In comparison to other studies in which two control 
commands are generated based on motor and arithmetic tasks 
with low classification accuracies results (Stangl et al., 2013), the 
results have shown potential for control command generation 
with high classification accuracy. Moreover four stage classifier is 
required if only one modality is used for four signals acquisition 
which may results in significant decrease in accuracy (Vuckovic 
and Sepulveda, 2012). Using the current setup, the same results 
with better accuracy are achieved: The accuracies of "Forward 
vs. Stop" and "Backward vs. Stop" trials were 80.2% and 83.6%, 
respectively, due to the selection of oxy-hemoglobin (HbO) as a 
classification feature for both control commands. This is the first 



time that four intended movements are decoded using a hybrid 
BCI. Wolpaw and McFarland (2004) and the previous two hybrid 
EEG-NIRS studies (FazH et al, 2012; Kaiser et al, 2014) have 
shown discrimination of two signals that can be used to gen- 
erate two control commands whereas in this paper four control 
commands have been generated. 

The generated commands can be used for those people who 
cannot perform motor imagery (Vidaurre and Blankertz, 2010) or 
in such cases that the patient cannot touch any machine directly, 
for instance, prosthetic legs, working with remote controlled 
devices, etc. Naseer et al. (2014) have shown binary decision 
decoding for rehabilitation, whereas in this research the four 
control commands can be associated to four different decisions. 
Moreover, the previous hybrid EEC based researches for rehabili- 
tation have shown the use of P300 and steady state visual evoked 
potentials (Li et al., 2013; Xu et al., 2013) to generate four con- 
trol commands based on reactive tasks. As motor imagery and 
motor execution activate the same brain area (Beisteiner et al, 
1995; Porro et al, 1996), the same goal can be achived by this 
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Table 2 | Classification accuracies: four control command accuracies 
of 12 subjects for left vs. stop, right vs. stop, forward vs. stop, and 
backward vs. stop. 



Subject 


Left vs. 
Stop (%) 


Right vs. 
Stop (%) 


Forward vs. 
Stop (%) 


Back vs. 
Stop (%) 


1 


98.4 


98.7 


81.1 


86.5 


2 


99.8 


99.2 


79.1 


84.1 


3 


93.2 


94.7 


81.2 


85.4 


4 


97.7 


96.5 


80.1 


80.8 


5 


98.1 


88.2 


81 .2 


86.9 


6 


91.1 


99.5 


84.2 


86.6 


7 


99.2 


98.1 


77.2 


81.1 


8 


98.0 


98.2 


83.1 


87.1 


9 


91.2 


90.4 


80.2 


83.1 


10 


85.0 


84.1 


74.1 


76.3 


11 


94.5 


95.2 


82.8 


86.1 


12 


90.1 


94.0 


78.2 


79.3 


IVIean 


94.7 ± 4.6 


94.7 ± 4.8 


80.2 ± 2.7 


83.6 ±3.5 



research using active tasks. Also, it is known that fNIRS can 
decode muhiple signals from the same brain area: for instance, 
mental counting and music imagery (Power et al, 2012) and pic- 
ture imagery and mental arithmetic (Naito et al., 2007) from the 
same prefrontal cortex. Therefore, there is a potential that the 
total number of commands in the current EEG and fNIRS config- 
uration can be increased to up to seven. Further research can be 
carried out using advanced adaptive filtering techniques (Rehan 
and Hong, 2013), optimal feature sets, and/or combined EEG- 
NIRS features to achieve better classification results that can be 
used by patients with lower limb disorder to control wheelchair 
or prostheses. 

CONCLUSION 

In this research, a hybrid fNIRS and EEG configuration for decod- 
ing four movment commands was proposed. The NIRS setup was 
used to decode the prefrontal activities based on hemodynamic 
changes of HbO using mental arithmetic and mental count- 
ing as tasks. The EEG response detected from the motor cortex 
region was used to decode other two direction commands. Both 
modalities were synchronized to obtain control commands at the 
same time. The results of classification accuracies were highly 
encouraging and certainly wUl prove fruitfully applicable to BCI 
systems and purposes. The use of wirelss systems and transla- 
tion of the control commands into machine codes will enable 
effective control of a robotic system suitable for rehabilitation 
purposes. 
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